DOCUMENT RESUME 



ED 041 041 



TM 000 006 



AUTHOR 

TITLE 

INSTITUTION 
PUB DATE 
NOTE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Karl, Marion C. 

An Example of Process Evaluation. 

Allegany County Schools, Pittsburgh, Pa. 

Mar 7 0 

12p.; Paper presented at the joint annual meeting of 
the American Educational Research Association and 
the National Council on Measurement in Education, 
Minneapolis, Minnesota, March, 1970 

EDRS Price MF-$0.25 HC-$0.70 

^Action Research, Concept Formation, Critical 
Thinking, Cultural Awareness, Cultural 
Interrelationships, ^Curriculum Evaluation, 
Evaluation, Experimental Programs, Library Skills, 
^Program Evaluation, Program Improvement, Social 
Studies, Student Evaluation 
Allegheny County Pennsylvania Schools, 

*Intercultural Understanding Project, Watson Glazer 
Test of Critical Thinking 



ABSTRACT 

The inappropriateness of standard experimental 
research design, which can stifle innovations, is discussed in 
connection with the problems of designing practical techniques for 
evaluating a Title III curriculum development project. The project, 
involving 12 school districts and 2,500 students, teaches concept 
understanding, critical thinking, and research skills through the 
medium of a world cultures course. Practical evaluation techniques, 
which rely heavily on subjective assessments by the project staff, 
teachers, school administrators, and students, were developed. 
Although in many instances it was necessary to modify standard 
research procedures, nonetheless product, process, and student 
evaluations are made and areas of needed revisions are revealed while 
the project is actually proceeding. The project objectives and survey 
forms are included. (Author/ES) 
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An Example of Process Evaluation 
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Marlon Karl 



From reports I have read from other title III projects and articles In some 
of the professional journals. It seems that In general, evaluators; are overlooking 
in opportunity to have real Impact on the directions of educational change. Too 
often evaluation Is limited to final result assessment and process evaluation Is 
neglected or absent. We will all agree that product evaluation Is necessary, but 
It 1$ process evaluation that points to the areas where revisions are necessary 
while the opportunities for revisions still exist. 



There are many reasons few attempts at process evaluation are made. No neat, 
scientific research design Is probable. Assumptions Inherent In the models for 
statistical analysis of the data must be violated. The researcher doesn't have a 
tenth .of the control he would like. Subjectivity is rampant. At times, one feels 
he Is attempting a task as impossible as analyzing the water at a given polnc in 
a moving stream. 



A dweller In educational research's ivory tower can cite such reasons for 
refusing to dirty his hands In such messy projects and convincing others that noth- 
ing valid can come of work which necessarily produces tenuous, inconclusive find- 
ings. Granted, there Is little one can say with certainty, but clues do become 
evident and trends do appear. 

Is It not better to make the effort to extract what we can while projects 
are still fluid than to wait for more reliable findings about which, too often, 
nothing can be done? Suggestions for further research have become an accepted 
adjunct to research reports, but how often does anyone take up the challenges 
they present? Funds dry up, boredom sets in, a team disperses, or a new and 
more exciting project looms on the horizon. While a project is in operation, 
the time. Incentive, energy and power to Improve ard available. Once it Is fin- 
ished, and Its faults and shortcomings are ennumerated, it often dies and what 
was once a marvelous idea that caught the interest and Imagination of a funding 
agency, staff and participants, is burled. 

We, of the Intercultural Understanding Project, believe a concentration on 
process evaluation can, to a large degree, prevent this from happening. The 
belief that the evaluation of what goes on in the real world of the classroom 
cannot be truly assessed by standard experimental research design Is not original 
with us. Last year in Educational Researcher , Guba went so far as to suggest 
that evaluation and research are two distinct and separate entitles and that It 
Is Inappropriate to apply rigid research models to evaluation problems. James 
Finn makes a strong case against "Institutionalization" of Title III evaluation 
In the December, 1969 Issue of Educational Technology . He feels that attempts to 
over-standardlze evaluations forces projects into in-flttlng molds which will 
stifle the very Innovations Title III Is meant to create. 

Few, If any, projects or researchers have managed to solve the dilemma pre- 
sented by the demand and necessity for meaningful evaluations of classroom pro- 
grams. No matter how sophisticated our research design tools we cannot begin to 
measure all the myriad facets of a learning experience of which we are aware; and 
all of us are fairly certain that there are just as many variables of which we are 
not aware. 
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What can b« done? Right no w, at least, we can't solve the problem, but we 
can perhaps lessen It by using a variety of approaches to evaluation as suggested 
by Stufflebeam (1969), One of the approaches Is process evaluation, which nd de- 
fines as "the way to Identify and monitor, on a continuous basis, the potential 
sources of failure In a project". It Is this aspect of evaluation that I address 
qyself to today. 

Before beginning, an Introduction to the project Itself should provide the 
common ground necessary to a joint consideration of both the Inappropriateness of 
banking entirely on experimental techniques and the relative values of process 
evaluation In work of this kind. 

The Intercultural Understanding Project Is a Title III Curriculum program 
now In Its second year of operation. Its primary function Is to develop Innovative 
materials for use In secondary world cultures classes. The emphasis Is on the 
teaching of research skills, concept understanding, and critical thinking rather 
than on the accumulation of a body of factual information. Our rationale Is that 
tomorrow's facts may be unknown today, but that the techniques for discovering 
and Interpreting these facts can be taught. Hence, If we can teach students how 
to question, how to read critically, how to gather and assess sources of Information 
and how to organize and analyze data, we will have provided them with the tools nec- 
essary for examining any culture. 

Opportunities to engage In these kinds of activities are provided through the 
complete teaching packages which we assemble for each of the cultural areas with 
which we are concerned. Included In the package are teacher lesson plans, student 
readings and activities, supplementary readings for teachers, slide-tape series, 
transparencies, an area bibliography, a bibliography of the works of the outstand- 
ing writers and suggested student evaluation devices. 

We also conduct frequent In-service sessions for participating teachers. 
Methodological philosophies and strategies are discussed at some, while others are 
devoted to new unit orientation. These meetings are meant to help teachers gain 
the necessary background to teach these new materials effectively. 

During the first year the program was operational, 1,48? pupils In grades 9 
through 12 were enrolled. They were taught by 12 teachers In 8 different suburban 
school districts In Allegheny County, Pennsylvania. Design problems appeared Im- 
mediately. The majority of students were from middle or upp*r middle class homes 
and of average ability. About 60% of them were homogeneously grouped and 40%, 
heterogeneously grouped. The impossibility of assembling comparable control groups 
becama more and mre apparent as the year wore on and although control groups were 
used, only the large numbers Involved saved the data from being completely Invalid. 

First of all, experlnantal teachers were In the program for a variety of 
reasons. Some were genuinely Interested In the objectives of the program and 
volunteered, some were told by their acini nlstra tors to Join the project, others 
were Interested more In furthering their own status than In the effect on their 
Students and still others, merely bored with their present course, There was no 
Why to find out all the factors that motivated every participating teacher. Second- 
ly, the standards for ability grouping varied from school to school. Since all 
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schools do not use the same Instruments for determining I.Q., even a search of 
permanent records wouldn't have established a uniform equation, nor did we have 
the time, money, or staff to administer one ourselves. Other areas which made 
control selection difficult were the assignment of socio-economic labels to 
conwunltles with wide ranges of variables, the subtleties of administrative phi- 
losophy, and the diverse quality of educational facilities. 

Half of the schools were able to provide their own control teacher, eliminate 
ing some of the contamination mentioned above. We had one teacher, however, who 
was her own control amd she found It Impossible not to share anything she thought 
particularly good In the experimental material with her control students. Another 
teacher re-classified his ability groups three-quarters of the way through the year. 

The design problems we were faced with are not atypical. Many researchers who 
wish to work In the public schools are there without the blessing of the teachers 
or the administrators. They must structure their work so as not to Interfere with 
the existing programs and policies If they expect cooperation and they must be flex- 
ible enough to adjust to the many deviations from the Ideal experimental situation. 

You cannot randomly assign pupils to treatments nor teachers to programs. 

Testing cannot always be accomplished at the optimum time or with standardized pro- 
cedure. How do you cope with the fact that a substitute teacher administered the 
test and answered some unremembered student questions? Or what do you do If the 
administrator says your attitude test might offend the more conservative element 
of his community and tells his teachers to have their students skip Items 6, 11, and 
31? How do you compute pre-and post- test correlations if the teacher loses the list 
of student numbers he assigned at the pre-test session and assigns new ones for the 
post-test? I could go on to cite pages of examples, but I'm sure you get the idea 
of what can and does happen to the best laid plans of researchers in the public 
schools. 

Without a monitoring system, which process evaluation is, the product evalua- 
tion contamination mentioned above may never come to light, and the "hard data" we 
have so much confidence In may really be pretty soft after all. The value of process 
evaluation Increases In proportion to the decrease In usable hard data. 

Everyone concerned with the Intercultural Understanding Project Is Involved In 
our process evaluation: teachers, administrators, students and staff. The most 

valuable members of this team are the participating teachers. Each week they send 
to the project office their comments on the week's work. They note revisions need- 
ed In the material, novel testing devices they have used, additional resources they 
found helpful. Interesting discussion topics which arose. Innovative presentation 
methods tried, problems they have had and areas where they need help. At periodic 
in-service meetings, staff and teachers review these sheets and the specific problems 
of each lesson are discussed. Student response to readings, visuals and activities 
ire surveyed to decide what went well and what didn't. We don't look for unanimity, 
but we have found that If a few teachers express difficulty with something, the others 
have nbt found It one of the better parts of the lesson. Sometimes trouble spots 
can be cleared up simply through better directions on the use of the materials, while 
other times, an entire lesson must be rewritten or discarded. It Is Interesting that 
the Initial unit we prepared on Japan was acclaimed by numerous experts as the best 
set of materials they have ever seen on Japanese culture, but It did not work In the 
classroom. It had to be completely rewritten, for after all, the final criterion Is: 
Does It do the job for students? 
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There have been times when an evaluation sheet sent In by a teacher has point- 
ed up some flaw In the materials that the staff deems so serious that a correction 
Is sent out Immediately. Since all teachers do not teach at the same rate, this 
allows the slower moving ones to avoid the flaw entirely. 

Students, too, are asked to react to specific lessons. Usually each teacher 
collects five student critiques of each lesson, making sure that all his students 
participate eventually. These reaction sheets are filed and used In conjunction 
with the teachers 1 when revisions are made. 

Students also participate In the actual writing of the materials. The first 
summer the project was In operation seven high school students were employed to 
help In the development of the Introductory unit In which the American Teenage 
Subculture Is explored. It was felt that students cannot be expected to under- 
stand other cultures without first Investigating their own. This approach proved 
to be a highly motivating way to have students gain the necessary background for 
the year*s work. We find students can make a worthwhile contribution to curriculum 
development and strongly recommend their use under properly supervised conditions. 
During the second summer, ten students were employed. They assisted In the revision 
process and reviewed new material. 

The combination of student reaction sheets and student Involvement In the 
actual development of materials has provided a basis for a part of our evaluation, 
which we found truly meaningful. 

Administrators are another source of evaluation personnel. They are con- 
sulted about project technical problems and are asked to report on classroom ob- 
servations made on their own. Having administrators as part of the evaluation 
team has resulted In their supportive interest In their teachers and an under- 
standing. on their part, of the objectives and accomplishments of the project. 

This Is a great asset when prospects for continuing the project beyond the t federal 
funding period are explored. 

The staff coordinates and analyzes all the evaluation feedback from teachers, 
students, administrators, as well as their own observations. They make the final 
decisions on revisions and new directions the project will pursue, using every 
source available to them. 

Of course we supplement subjective forms of evaluation with objective ones. 

They work well together and compliment each other. For Instance, the computer- 
ized analysis of the final skills test., patterned partially after the Watson- 
Glazer Test of Critical Thinking, Indicated that the average ability student did 
significantly better with the materials than did either the high or low ability 
groups. The analysis gave us no clue as to why this was true. Through staff 
visitations to classrooms and teacher discussions at In-service meetings. It was 
discovered that high ability students had a difficult time adjusting to the ab- 
sence of memorizable material that could be regurgitated on tests. On the other 
hand. It was brought out that low ability students, while still scoring at the 
bottom of the test scale, were contributing more to project classes where all 
logically supported opinions find equal acceptance. 
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Another aspect of this same analysis revealed that 10th and llth grade did 
significantly better with the materials than did the twelfth grade. ‘"Weekly 
evaluation sheets had already alerted us to the fact that by 12th grade students 
have developed research strategies of their own and are relatively Insensitive to 
the subtle differences presented In the new material. (Besides, the school which 
was using our materials In 12th grade had a requirement that 12 weeks of economics 
be squeezed Into the World Cultures course somewhere). 

We also did an Item analysis of the skills test. Through It, we discovered 
that most of our students could not Interpret graphic material fully or well. 

They also had difficulty In recognizing logical Inferences, using the card * 
catalogue, and appraising the legitimacy of Information sources. Because of these 
findings, more of these kinds of activities have been built Into the revised 
materials. For example, a lesson on the economy of Japan lent Itself to an 
exercise In Interpreting graphs and charts. Instruction 1 n various graphic formats, 
descriptions of the types of data most appropriate to each, and the Interpretations 
Indicated, preceded the exercise. Although the material for the lesson was gather- 
ed by the researcher, the curriculum specialist Incorporated It Into the unit as 
a natural component. 

Pre- and post- measures were administered to determine student's ability to 
distinguish fact from opinion and to detect changes In their attitudes towards 
other races, religions, and cultures. We were satisfied with the results of the 
Fact-Opinion Test. The attitude test data, however, revealed that while experi- 
mental students were more open-minded towards people of foreign cultures, the 
control students were more open-minded towards minority groups In our own country. 

This finding will Influence revisions made In the Introductory unit. 

Necessarily, the main focus of our evaluations Is the material we produce since 
these are our raison d'etre. In attempting to assess them thoroughly, we must make 
judgments about their effect and affect on students and teachers. By approaching this 
assessment from many different angles we try to compensate, at least In part, for 
the lack of control necessary to a more scientific Investigation. By making the 
assessment a continuing process, revisions are made while It Is still possible to 
learn from them. We concur with Cronback's (1963) statement: "Evaluation used 

to Improve the course while It Is still fluid contributes more to the Improvement 
of education than evaluation used to appraise a product already placed on the mar- 
ket." 



This glimpse Into the evaluation activities carried on at our project hope- 
fully has given you a concrete example of process evaluation that Is usually dis- 
cussed In the abstract by Its proponents. There Is no neat scientific research de- 
sign; statistical assumptions are violated; little control Is possible; subjectivity 
1$ rampant, but at least for the 2,400 students, 20 teachers and 12 school districts 
we are serving this year, we are having an Impact on the direction of educational 
change. 
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Students will exhibit an Increased Interest In intercultural (international) affairs as displayed 
by Increased use of school library facilities , Increased discussion and participation In class- 
room activities, and by attending non-credit, after-school programs related to areas of World 
Cultures interest. 
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Students will develop an understanding of the various Institutions found in cultures (kinship, 
religion, education, government, and economy), their role and their effect on contemporary 
world affairs. This understanding will be exhibited by naming the Institutions, comparing 
or contrasting their functions with similar Institutions in other cultures, by noting changes 
In institutional functions t j> contemporary decisions on foreign and domestic policy through 
discussion und/or written analysis. 



3. 



Student attitudes will be changed In a positive direction as they relate to members of other 
races, religions, or cultures as evidenced by performance on pre/post attitude tests, classroom 
interaction with foreign and Peace Corps personnel, and with class members of differing 
backgrounds. 



The students will develop skills In using the techniques of the social scientist, such as data 
collection, analysis and evaluation of data, formulation of hypotheses, projecting solutions, 
validating hypotheses based on available evidence, and supporting his conclusions In oral 
and/or written form. 
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5. Students will examine a wide variety of viewpoints prior to forming opinions and give evi- 
dence of this by documentation of varying viewpoints; by analyzing and verbalizing his 
own point of view in terms of the experiences he brings to the learning situation; by ex- 
amining two or more cartoons, articles, etc., pertaining to the same topiq and Identifying 
the author's frame of reference; and by conducting research into thp background of 
authors to determine possible bases for their bias. 
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6. Teachers of World Cultures courses will be retooled to enable them to use an interdisciplinary 
approach in developing understanding of intercultural concepts, as demonstrated by the 
teacher's ability to verbalize areas of concern for each of the social sciences in the analysis 
of a culture, by demonstrating competence in their ability to show interrelationships of the 
social sciences in units developed during their area In-depth study, and by drawing analogies 
to meaningful life situations of their students. 
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7. 



Packages of materials will be developed and selected to meet individual needs of students 
at their various achievement levels by determining student needs indicated by past and 
present performance, by screening an q evaluating commercially-prepared materials (print- 
ed and non-printed) as needed, by developing simulated materials at varying achievement 
levels, and providing tapes of Important speakers or foreign resource persons for students 
in need of additional help, review, or those who may have.been absent 
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8. 



Students will be provided with a shared real-life contact, in the classroom, with members 
of other cultures or with people who have been closely associated with other cultures by 
providing foreign resource personnel. Peace Corps volunteers, and State Department 
personnel. 
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TEACHER'S WEEKLY EVALUATION SHEET 



Name Week From t to 



INTERCULTURAL UNDERSTANDING PROJECT 
Allegheny County Schools 
100 Ross Street, Fourth Floor 
Pittsburgh, Pennsylvania 15219 

WEEKLY REPORT SHEET 



Please consider the following topics and write a line or two in the areas appli- 
cable to your week’s work. 

1. Novel presentation methods used 



. . . f _ ^ — - - i ■■■ i — -- ... 

2. Noteworthy topics and discussions 



3. Non-project supplied materials you have found helpful _ 



A, Student reactions to any or all of the above 



2 •" 

5. Strengths and weaknesses of the program 



6. Problems you have had 
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T 
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7. Recommendations tor revisions 



8. Areas where we can be of help 



1 w ' 1 * » ■ ■■ — " 1 f ■ ■ ■ I. ,,, 

9. Unique testing or evaluation methods used 



10. Enclose samples of work or testing materials you feel might Interest others 
in the project. 
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STUDENT LESSON SURVEY Lesson #_ 

> 

This survey form Is designed to lUev you to express your reactions to the various 
lessons on Africa. It Is not pecessary to sign your name* Please give each category 
your serious consideration. This la your chance to Influence curriculum development. 

All questions may not be applicable to every lesson . 

1. Do you believe the lesion helped you: 

a. Understand Africa and the problems of Africans better? 

Yes H o N ot Applicable 

b. See relationships between problems In Africa and those In other cultures. Including 
your own. 

Yes N o N ot Applicable 

c. Distinguish similarities and differences between African culture and other cultures. 

Yes N o Not Applicable 

Comments: . 



2. Do you believe the readings were: 

a. Interesting Boring Sat is factory 

b . Too Long Too Short About Right 

T" ■ 

N 

c. Too Difficult Too Easy About Right 

3. If applicable, please comment on the reading (s). Did you like or dislike It (them)? 
State Why. 






4. If applicable did you find the slides: 

a. I nformative N ot Helpful 

b. Interesting __Dull All Right 

5. Was there sufficient material available, either In the readings or In the library, to 
complete the activity? 

Y es N o 

6. Do you have any suggestions to Improve the lesson — readings, presentation In class, etc 
(List on other side). 



IlfTERCULTUBAL UNDERSTANDING PROJECT 
Allegheny County Schools 
100 Ross Street, Fourth Floor 
Pittsburgh, Pennsylvania 15219 

LIBRARIAN SURVEY 

The purpose of this questionnaire is to note any differences In the use of 

library facilities and materials by the students in the Intercultural Understand- 
ing Project World Cultures clasges, as compered to last year. Tour cooperation 

In completing this form will be greatly appreciated. 

1 Is /are the participating teacher (s) from 

your school. 

1. In general. World Cultures students spent m ore 

time in the library as last year. ~ 

2. The number of books checked out by these students 
was approximately the same. 

• 

3. The demand for periodical material increased decreased was about 

the same. 

A. World Cultures students were scheduled into the library for entire class periods 

more frequently less frequently a bout the same number of times as 

last year. 

5. The library was able to supply the needs of these students well adequate- 
ly less than adequately. 

6. Please note any areas where you feel you need additional materials. 



7. Have you purchased any materials this year because of the new World Cultures 
program? Yea No 

If yes, note what you haVe bought and why. , 



_less about the same 

increased decreased 




Please return this form to: Mrs. Marion Karl, Intercultural Understanding Project, 

100 Ross Street, Fourth Floor, Pittsburgh, Pennsylvania 15219. 



Form Uied to Record Staff Visitations to Project Classes 



Schoo l T eacher 

Visitation bvs 

Unit : L esson: 

Class observed: E3 yes ZZ7 no 

Free period discussion: £7 yes JC7 no 

COMMENTS: 



PROBLEMS: 



RECOMMENDATIONS: 



FOLLoW UP: 



